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ABSTRACT 



Individual program evaluation studies have the potential to 
accumulate of very valuable knowledge and practical wisdom. Drawing on "The 
Case for Pragmatic Psychology" (D. Fishman, 1999), this paper contends that 
much of this wisdom is being thrown away because of the positivist paradigm 
used in determining which knowledge should be archived through publication 
and which should be discarded. It is suggested that a way of retaining the 
wisdom of individual evaluation studies is to implement a new "pragmatic 
psychology" paradigm within which individual program evaluations are 
documented in peer reviewed, systematic case studies that are electronically 
published in searchable, cumulative databases. (Contains 12 references.) 
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Individual program evaluation studies have the potential to accumulate very valuable 
knowledge and practical wisdom. Based upon the author’s recent book. The Case for 
Pragmatic Psychology (Fishman, 1999), this paper contends that we are presently throwing 
away much of this wisdom because of the positivist paradigm we use in determining which 
knowledge should be archived through publication and which should be discarded. The 
paper further argues that a way of retaining the wisdom of our individual evaluation studies 
is to implement a new, “pragmatic psychology” paradigm, within which individual program 
evaluations are documented in peer reviewed, systematic case studies that are electronically 
published in searchable, cumulative databases. 



*Paper presented at the annual conference of the American Evaluation Association, 
Orlando, FL, November 6, 1999. For further information, contact: Daniel B. Fishman, 
Ph.D., Graduate School of Applied and Professional Psychology, Rutgers University. 
Mailing address: 2 Clinton Avenue, Dobbs Ferry, NY 10522. Email: dfish96l98@aol.com . 
Fax: 603-917-2567. 
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Ah. Success! 

Consider a hypothetical example coming out of the recent, tragic shooting at 
Columbine High School in Littleton, Colorado and the national concern about violence 
presentation programs in the schools. Suppose there is a “Reach Out High School” that is in 
a predominantly white suburban section of a Rocky Mountain city that shares some of the 
larger gun culture that is found in other Western states like Colorado. Also suppose that the 
types of students attending Reach Out High are essentially the same as those who attend the 
other high school that serves the same geographic area — which I will call “Traditional High 
School.” 

Suppose I conduct an in-depth program evaluation of Reach Out High's and of 
Traditional High's violence prevention programs encompassing both (a) an ethnographic- 
style, qualitative study, and (b) a series of quantitative measures, including number and 
severity of incidents involving suicidal or homicidal violence, number of disciplinary 
incidents, academic achievement, attendance rates, ratings by teachers of school safety and 
staff morale, and ratings by students and parents about the safety and overall satisfaction 
within the school. 

And suppose that in my evaluation I find very clear-cut evidence -- both qualitative 
and quantitative — that violence prevention, achievement, and morale in Reach Out High is 
dramatically better than that in Traditional High. This holds true across the variety of 
measures and factors I assessed. My evidence is particularly strong in that the Reach Out 
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High results are impressive both in absolute terms, on their own, and comparatively, in 
terms of showing a dramatic superiority over those of Traditional High. 



Positivism and Case Studies Don’t Mix. 

The staff, board, and parents of Reach Out High will most probably be avid 
consumers of my results. But what about others in the program evaluation profession? Or 
professional educators? How will they obtain systematic, professional knowledge about the 
successful violence prevention program? 

The usual way one would think results are disseminated is through the published 
program evaluation literature. What would be required for my evaluation to be published? 
While program evaluation is from one perspective a strongly applied field, at the present 
time it continues to be dominated by positivist social science, the “received view” (Guba & 
Lincoln, 1994; Greene, 1994). This paradigm searches for general knowledge. This means 
that when it comes to publication, the basic unit of knowledge is the experimental or quasi- 
experimental study that tests theoretical hypotheses across groups of subjects (Fishman, 
1999). When this paradigm is brought to bear in the area of program evaluation, the result 
is a set of publication criteria that emphasize studies with themes like: (a) confirmation of a 
theory of social causation, (b) confirmation of a theory of social program functioning, (c) 
development of a new theory of program evaluation, or (d) development of a new 
methodological set of program evaluation procedures. 

Back to my particular study results. On one hand, the results would seem to be of 
great practical value and relevance for other schools with similar types of students. On the 
other hand, due to present publication policies -- and the positivist paradigm upon which 
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they're based — all these results will be basically lost to the profession and to the future. 
This is because the value of my program results is not related to theory-testing or an 
innovative methodology per se. 

Rather, the value of my study is related to the results that I found: the 
documentation of an individual school with an outstandingly effective violence prevention 
program, that is, documentation of a school that works (Fishman, 1999). Of course, maybe 
I can obtain publication for my study because of the unique situation created by all the 
national publicity over the Colorado shooting. But generally, there are no publication 
outlets for these types of program evaluations. 

Now think of all the other similar evaluation studies that are in essence lost to the 
published knowledge base of our field. From the perspective of “pragmatic psychology,” 
this is a terrible waste (Fishman, 1999). 



Needed: A Good Construction Company 

What I’m calling “pragmatic psychology” is one of a variety of alternatives that 
today challenge the dominant positivist epistemology. Positivism emphasizes the search for 
general, objective knowledge through experimental theory-testing in laboratory-like 
settings. The challenges to positivism have grown at an increasing rate in recent years. 

These challenges go under a variety of names, such as "postmodernism," "social 
constructionism,” “cultural criticism," "hermeneutics,” and “qualitative research” (Denzin & 
Lincoln, 1994). I use the term "social constructionism" to encompass them all. 

While there are important differences among these frameworks, they all contrast 
themselves to positivism. They assume that reality is, to a large extent, "constructed" or 
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"invented" by individuals and groups. This construction takes place as a function of 
particular personal beliefs and historical, cultural, and social context. Thus "social 
constructionism" argues against positivism's claim to achieve fundamental, objective, 
general knowledge about the world through the natural science method. Rather, the social 
constructionist contends that such knowledge is always limited by the subjective and 
cultural context of the knower. And so the "eternal" truths promised by natural science are 
limited to particular cultural, historical, and social perspectives and contexts (Fishman, 

1999; Gergen, 1985). 

Getting Down To Cases 

In line with the tenets of social constructionism, pragmatic psychology views 
practical knowledge-in-context as crucial to its value. More specifically, in pragmatic 
psychology this knowledge-in-context consists of the systematic case study of individual 
human service programs. The research focus in a pragmatic study, then, is not upon the 
testing of a general theory. Rather, it is upon the condition of one or more particular clients 
— be they an individual, group, organization, or community - in a particular program. The 
focus is on first, what the client's presenting problems and associated goals are at the 
beginning of the program, and second, whether during the course of the program, the client 
is helped to achieve these goals. So, within pragmatic psychology, case studies of particular 
programs constitute the knowledge foundation of present program planning and evaluation 
(Fishman, 1999). 
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From Single Case to Database 

The pragmatic case study actually draws from both the positivist, natural science 
and social constructionist traditions. More particularly, it combines (a) standardized, 
quantitative, "performance indicator" measures of presenting problems and outcomes, 
associated with the positivist tradition (e.g., Rossi & Freeman, 1999); and (b) qualitatively 
elaborated case formulations and process descriptions, associated with the social 
constructionist tradition (Denzin & Lincoln, 1994). 

Crucially, the pragmatic case study addresses what might be called the "idiographic" 
problem — the single case study's lack of generalizability of its programmatic and evaluative 
knowledge because of its N of 1 . The pragmatic case study method calls for organizing case 
studies of programs with similar target goals into computerized databases (Fishman, 1999). 

For example, consider a shelter program for the homeless in a very large city like 
New York, or a violence prevention program in a high school like Reach Out, discussed 
above. A write-up of either such a case is limited by the number of case situations in the 
future to which it will particularly apply. This is because large contextual differences can 
occur between this case and any other case that is randomly drawn out of a heterogeneous 
case pool. However, as cases in the database grow, they begin to sample a wide variety of 
contextually different situations in which the target problem can occur. So, as the number of 
cases in the database rise, the probability increases that there exist specific cases in the 
database that are particularly relevant to an ongoing target case. 

In order to decide whether a particular completed case will generalize to an 
ongoing target case, it is necessary to have a detailed, qualitative, "thick" description of the 
conditions and context of the completed case. A good model for pragmatic psychology in 
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this regard is judicial law (Bromley, 1986). The basic unit injudicial law's knowledge base is 
the individual case. Thick description in the case includes transcripts of everything that was 
said and presented during the case, the judge's or jury's decision and the rationale for that 
decision, and all similar documents from any appeals associated with the case. The overall 
process in a judicial case is to view the unique constellation of the case's facts in the context 
of a particular legal conceptual framework. This legal conceptual framework is defined in 
most U.S. law by both deductive, legislated principles and inductive, English common law 
principles. The parallels to this legal framework in a case study of a human service program 
are positivistically derived social science theory and grounded social science theory (Strauss 
& Corbin, 1990), respectively. Both of these provide the conceptual base for developing a 
human service program and should be open to change and revision based upon case results. 

There’s Gold in Them Thar Databases 

The type of reasoning employed to find features from past cases that are relevant to 
present cases — "learning from experience" — has been incorporated into a branch of 
artificial intelligence called "case-based reasoning” (Gentner & Holyoak, 1997; Kolodner, 
1997). And this field provides a paradigm for developing the "single case to database" 
concept. As in other computer databases, each case would be categorized in various ways, 
creating the capacity to search for and access particular types of cases. 

A practicing planner or evaluator, working with the problems, goals, and contexts of 
a particular program, could then access those case studies of other programs that best 
matched the target situation. Once found, these cases could provide guidance to the planner 
or evaluator in addressing their present target situation. This is similar to calling upon your 
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past experience to deal with a present situation. Or, consulting with colleagues about their 
experience. 

The difference is that the knowledge in the database would be more systematic and 
clearly thought through, since it would be professionally written out. You would also have 
the advantage of consulting with much larger groups of professionals, including people 
you’ve never met. In short, the database would magnify the breadth and efficiency of the 
professional networking process (Fishman, 1999). 

In addition to evaluation practitioners, researchers and theorists in the field could 
conduct comparative analyses across specific types of cases. These analyses could yield 
pragmatically focused generalizations about what intervention factors are associated with 
accomplishing certain types of goals in specific kinds of case situations, and also could 
contribute to grounded theory development. Since the full case study would be available, 
each factor and theoretical possibility would be explored in the context of the total service 
program. In other words, the exploration of a specific intervention factor would be 
embedded within the reality of the total intervention process, including the myriad of other 
variables of potential interest and relevance. 

Throughout, evaluation would be a crucial component of each case study in 
determining its outcome, in determining whether the case study is to be viewed as a model 
for other programs or as an object lesson in pitfalls to be avoided. 

The Nuts and Bolts of Rigor. 

For program evaluation case studies to merit publication, they must be systematic 
and rigorous in method. In The Case for Pragmatic Psychology, I propose an initial place 
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to start for such a method. My proposal integrates work drawn from the fields of traditional 
psychometrics, program evaluation, qualitative research, and community psychology to 
develop a working set of general guidelines for such a methodology that encompasses 
systematic case studies of any type of human service program. These guidelines are laid out 
in terms of the typical organization and content of a traditional group research study write- 
up: introduction, method, results, and discussion. For comparison, parallel items are 
included for the positivist group study and the ethnographic, hermeneutically focused case 
study (Lincoln & Guba, 1985; Guba & Lincoln, 1989). This comparison illustrates that the 
pragmatic paradigm is integrative, drawing methodological concepts and procedures from 
each of these other two paradigms. While space does not allow a full description of the 
guidelines, Table 1 presents a very brief summary of the positivist and pragmatic paradigms. 

As can be seen in Table 1, a pragmatic case study, or “PCS,” begins in the 
Introduction with a focus on client problems to be solved. The program’s initial guiding 
theoretical framework for conceptualizing the problems, generating goals for change, and 
creating strategies for attaining the goals is called the program’s “guiding conception.” In 
content, this guiding conception is typically both molecular and holistic, systems-oriented 
and organic, and inclusive of a large number of variables.. 

In the Method section, the PCS describes and discusses a number of issues 
concerning the case setting, such as the rationale for choosing the case study, the 
boundaries of the case, a description of the program’s services, and a description of the 
qualitative and quantitative measures to assess program process and client goal attainment. 
In addition are methodological controls to deal with first, issues parallel to those in a 
positivist study, such as construct validity, internal validity, and reliability; and second, 
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issues parallel to those in a hermeneutic study, such as the logical adequacy of grounded 
theory that might emerge from the study, the credibility of process descriptions, and 
“transferability,” that is providing a “thick” description from which generalizability can be 
derived by induction. 

Next is the Results section that outlines the quantitative results relevant to program 
impact, and the qualitative results relevant to program process, including any grounded 
theory that might emerge from the study. 

Finally, in the Discussion section, there is practical application to other individual 
cases, and implications for confirming the general usefulness of the guiding conceptions in 
the study, including implications for grounded theory development. 



Reinventing the Wheel: A Journal for Many Reasons 

In light of the above, I propose that an ideal way of pilot-testing the pragmatic 
psychology paradigm would be to create a new, electronic, peer-reviewed “Journal of 
Pragmatic Case Studies.” The journal would be structured like a wheel. The hub would 
consist of papers that address philosophical, theoretical, methodological, logistical, 
economic, political, and ethical issues in the development of insightful and useful, systematic 
case studies. Sample issues might include the epistemological basis of case studies; the 
practical problems of retaining as much context as possible while protecting confidentiality 
in a case study; and the refinement and improvement of methodological standards for 
guiding the peer-reviewed evaluation of case study manuscripts. 

The spokes of the wheel would consist of particular databases of case studies, with 
each database focusing on human service programs aimed at a common goal, such as 
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preventing school violence, facilitating successful transition from welfare to work, reducing 
recidivism in juvenile offenders, or reducing dislocation caused by factory closings. Multiple 
spokes are necessary because, on one hand, there is a need for focus within a group of case 
studies to enhance generalization to other case situations of the same type. On the other 
hand, there is a need for many areas to be included in the journal to demonstrate the 
generality of the case study method to the evaluation of all types of human service 
programs. 

Organizationally, one or more editors-in-chief and an advisory board in charge of 
the whole venture would closely supervise the journal’s hub area. Subject area editors 
would be appointed to head up each spoke. This arrangement would allow for an 
overarching unity to the journal, while accommodating the level of specialty needed in 
particular areas. This unity would include a common set of guidelines for what defines an 
acceptable pragmatic case study in terms of epistemological grounding and methodological 
rigor. 

It is important to note that a “Journal of Pragmatic Case Studies” could virtually 
incorporate into its databases case studies already published in other program evaluation 
journals, such as Evaluation and Program Planning. All that is needed is a common set of 
peer-reviewed methodological guidelines among the journals included in the virtual case 
study databases. 

Ultimately, the proposed “Journal of Pragmatic Case Studies” could provide a 
vehicle for judging the pragmatic case study method on its practical results, that is, on 
whether the existence of databases of systematic case studies would in fact help to improve 
the performance of social service programs across the spectrum of human service needs. 
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The Winner Is. . . 

In sum, I’m arguing that the development of pragmatic case study databases has 
four major beneficiaries. 

First, it provides recognition for the work of professional evaluators by offering a 
publication outlet that directly parallels what is actually done in practice. 

Second, it creates a whole new body of practical, “expert consultation” knowledge, 
for both evaluation practitioners and researchers alike. 

Third, it provides a resource for the development of grounded theory with two 
crucial characteristics: it takes into account the epistemological limits of our socially 
constructed world, and it is guided by the practical need to address today’s pressing 
psychosocial problems. 

Finally, pragmatic psychology’s promotion of rigorously documented case studies of 
programs that work should be welcomed by a public hungry for promising news about our 
capacity to address societal problems. 
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TABLE 1. TYPICAL ORGANIZATION^ CONTENT OF A RESEABCHREPORTIN THREE PARADK1MS 
Abstracted from Fishman, D.B. (1999). The case for pragmatic psychology. New York: NYU Press, pp. 156-163. 



Section of Article 


Positivist Group 
Study Paradigm 


Pragmatic Case Study 
Paradigm 


Hermeneutic Case Study 
Paradigm (derived from the work 
of Yvonna Lincoln and Egon Guba) 


INTRODUCTION: 
purpose of the study 


To test a theory. 


To solve client problems. 


To explore a conceptual 
problem, to evaluate a program, 
or to explore a policy option. 


type of 

conceptualization 

employed 


In content, the theory 
to be tested is typically 
molecular, 
mechanistic, and 
limited to a few 
variables. 


In content, the guiding 
conception for addressing the 
client’s problems is typically 
both molecular and holistic, 
systems-oriented and 
organic, and inclusive of a 
large number of variables. 


Theory emerges during the 
study and is typically holistic, 
systems-oriented and organic, 
inclusive of a large number of 
variables, and negotiated with 
the research participants. 










METHOD: 

important variables to 
be considered 


Settings. 
Measures. 
Procedures for 
theory-testing. 


Case context. 

Rationale for choosing 
the case study. 

Case boundaries and 
relation to other 
cases. 

Program description. 

Stakeholder values 
and goals. 

Qualitative measures 
to assess program 
process. 

Quantitative 
measures to 
assess client 
goal-attainment. 


Case context. 

Rationale for choosing 
the case study. 

Case boundaries and 
relation to other 
cases. 

Program description. 

Stakeholder values 
and goals. 

Qualitative measures 
employing the 
researcher in a 
participant observer 
role as a “human 
instrument.” 


purpose of study 
design 


To ensure logical 
adequacy of theory 
testing. 


To ensure logical adequacy 
of assessing program 
outcome and process, 
including the adequacy of 
relevant grounded theory 
emerging from the study. 


To ensure logical adequacy of 
the grounded theory emerging 
from the study. 


construct validity 


Validity of the study’s 
measures in 
operationalizing the 
study’s theoretical 
variables. 


Reasonableness, logical 
coherence, and sociopolitical 
fairness of the performance 
indicators that are used to 
reflect a program’s process 
and the goals of its 
stakeholders. 


Not applicable. 


validity within the 
study 


“Internal validity” of the 
causal processes 
within the study 


“Internal validity” of the 
functional processes within 
the study. 


Not applicable 



0 
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TABLE li CONTINUED. 



Section of Article 


Positivist Group 
Study Paradigm 


Pragmatic Case Study 
Paradigm 


Hermeneutic Case Study 
Paradigm (derived from the 
work of Yvonna Lincoln and 
Egon Guba) 


METHOD (CONT.) 
Credibility 


Not applicable. 


Convincing logic and 
reasonableness relating the 
components of the program 
studied - guiding conception, 
assessment, formulation, action, 
evaluation, and feedback. 


Establishing isomorphism 
between the constructed 
realities of case respondents 
and the reconstructions 
attributed to them. 


Applicability to other 
sites 


“External validity” - 
generalizing from 
sample characteristics 
by deduction. 


“Transferability” - providing a 
“thick” description from which 
generalizability can be derived by 
induction. 


“Transferability” - providing 
a “thick” description from 
which generalizability can be 
derived by induction. 


Reliability 


Showing repeatability 
of the measurement 
process. 


Showing repeatability of the 
measurement process. 


Not Applicable. 


Dependability 


Not applicable. 


Establishing that the process of 
the study can be reconstructed by 
a research auditor. 


Establishing that the process 
of the study can be 
reconstructed by a research 
auditor. 


Researcher objectivity 


Establishing a 
researcher’s “value 
neutrality” and the lack 
of “experimenter bias.” 


Establishing procedures for 
reducing the impact of the 
researcher’s values and 
conceptual preferences. 


Not applicable. 


Confirmability 


Not applicable. 


Assuring that data, interpretation, 
and outcomes in the research are 
rooted in contexts and persons of 
the case apart from the 
researcher. 


Assuring that data, 
interpretation, and outcomes 
in the research are rooted in 
contexts and persons of the 
case apart from the 
researcher. 










RESULTS: 


Presenting quantitative 
results relevant to 
theory testing. 


Presenting quantitative results 
relevant to program impact, and 
qualitative results relevant to 
program process, including 
relevant grounded theory that 
emerges from the study. 


Summarizing the “grounded 
theory” that emerges from 
the researcher’s participant 
observation role. 










DISCUSSION: 


Implications for 
general, nomothetic 
theory. 


Practical application to other 
individual cases, and implications 
for confirming the general 
usefulness of the original guiding 
conception of the program, 
including implications for 
grounded theory development. 


Idiographic lessons to be 
learned, both about 
grounded theory and about 
empowering stakeholders. 
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